Regularization vs. Relaxation: A conic optimization perspective of statistical variable selection

Variable selection is a fundamental task in statistical data analysis. Sparsity-inducing regularization methods are a popular class of methods that simultaneously perform variable selection and model estimation. The central problem is a quadratic optimization problem with an `0-norm penalty. Exactly enforcing the `0-norm penalty is computationally intractable for larger scale problems, so different sparsity-inducing penalty functions that approximate the `0-norm have been introduced. In this paper, we show that viewing the problem from a convex relaxation perspective offers new insights. In particular, we show that a popular sparsity-inducing concave penalty function known as the Minimax Concave Penalty (MCP), and the reverse Huber penalty derived in a recent work by Pilanci, Wainwright and Ghaoui, can both be derived as special cases of a lifted convex relaxation called the perspective relaxation. The optimal perspective relaxation is a related minimax problem that balances the overall convexity and tightness of approximation to the `0 norm. We show it can be solved by a semidefinite relaxation. Moreover, a probabilistic interpretation of the semidefinite relaxation reveals connections with the boolean quadric polytope in combinatorial optimization. Finally by reformulating the `0-norm penalized problem as a two-level problem, with the inner level being a Max-Cut problem, our proposed semidefinite relaxation can be realized by replacing the inner level problem with its semidefinite relaxation studied by Goemans and Williamson. This interpretation suggests using the Goemans-Williamson rounding procedure to find approximate solutions to the `0-norm penalized problem. Numerical exThe first author is supported by the Washington State University new faculty seed grant; the second author is partially supported by the Simons Foundation Award 359494; the final author is supported in part by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Applied Mathematics program under contract number DE-AC02-06CH11357. Hongbo Dong Department of Mathematics, Washington State University, Pullman, WA 99163 E-mail: [email protected] Kun Chen Department of Statistics, University of Connecticut, Storrs, CT 06269 E-mail: [email protected] Jeff Linderoth Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, WI 53706 E-mail: [email protected] 2 Hongbo Dong et al. periments demonstrate the tightness of our proposed semidefinite relaxation, and the effectiveness of finding approximate solutions by Goemans-Williamson rounding.